{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reparameterization Trick\n",
    "\n",
    "Inspired from the example given by Goker Erdogan http://gokererdogan.github.io/2016/07/01/reparameterization-trick/\n",
    "\n",
    "Suppose we have a Normal distribution parameterized by $\\theta$: \n",
    "\n",
    "$$x \\sim p_\\theta(x) = N(\\theta,1) = \\frac{1}{\\sqrt{2\\pi}}e^{-\\frac{1}{2}(x-\\theta)^2}$$\n",
    "\n",
    "We draw samples from it and then use those samples to find the minimum of: $J(\\theta) = E_{x \\sim p_\\theta(x)}[x^2]$. And we do so using gradient descent.\n",
    "\n",
    "### Score/Reinforce way\n",
    "\n",
    "We take a derivative of $J(\\theta)$ with respect to $\\theta$:\n",
    "\n",
    "$$\\nabla_\\theta J(\\theta)$$ \n",
    "$$= \\nabla_\\theta  E_{x \\sim p_\\theta(x)}[x^2]$$\n",
    "$$= \\nabla_\\theta \\int{p_\\theta(x) x^2 dx}$$\n",
    "$$= \\int{\\nabla_\\theta{p_\\theta(x)} x^2 dx}$$\n",
    "$$= \\int{\\frac{p_\\theta(x)}{p_\\theta(x)} \\nabla_\\theta{p_\\theta(x)} x^2 dx}$$\n",
    "$$= \\int{p_\\theta(x) \\frac{\\nabla_\\theta{p_\\theta(x)}}{p_\\theta(x)}  x^2 dx}$$\n",
    "$$= \\int{p_\\theta(x) \\nabla_\\theta{log{p_\\theta(x)}  x^2 dx}}$$\n",
    "$$= E_{x \\sim p_\\theta(x)} [\\nabla_\\theta{log{p_\\theta(x)}}  x^2]$$\n",
    "\n",
    "We can now use Monte Carlo estimate of this using samples:\n",
    "\n",
    "$$\\widehat{\\nabla_\\theta J(\\theta)} = \\frac{1}{N} \\sum_{i=1}^{N} \\nabla_\\theta{log{p_\\theta(x_i)}}  x_i^2$$\n",
    "\n",
    "Substituting expression for $p_\\theta(x)$ from above and taking a log followed by gradient wrt $\\theta$, we get:\n",
    "\n",
    "$$\\widehat{\\nabla_\\theta J(\\theta)} = \\frac{1}{N} \\sum_{i=1}^{N} (x_i-\\theta)  x^2_i \\;\\;\\;\\; \\text{...(1)}$$\n",
    "\n",
    "\n",
    "### Reparameterization Trick\n",
    "\n",
    "Let: $$x = \\theta+\\epsilon \\;\\;\\;\\; {\\text{where}} \\;\\;\\; \\epsilon \\sim N(0,1)$$ \n",
    "\n",
    "This reparameterization leaves the distribution: \n",
    "\n",
    "$$p_\\theta(x) = N(\\theta,1)$$\n",
    "\n",
    "Let us now again calculate $\\nabla_\\theta J(\\theta) = \\nabla_\\theta E_{x \\sim p_\\theta(x)}[x^2]$\n",
    "\n",
    "$$\\nabla_\\theta J(\\theta)$$ \n",
    "$$= \\nabla_\\theta  E_{x \\sim p_\\theta(x)}[x^2]$$\n",
    "$$= \\nabla_\\theta  E_{\\epsilon \\sim N(0,1)}[(\\theta+\\epsilon)^2]$$\n",
    "As Expectation does not depend on $\\theta$, we can move the gradient inside without getting in the `log` issue as in previous method:\n",
    "\n",
    "$$= \\nabla_\\theta \\int{p(\\epsilon) (\\theta+\\epsilon)^2 d\\epsilon}$$\n",
    "$$=  \\int{p(\\epsilon) \\nabla_\\theta(\\theta+\\epsilon)^2 d\\epsilon}$$\n",
    "$$=  \\int{p(\\epsilon) 2(\\theta+\\epsilon) d\\epsilon}$$\n",
    "$$= E_{\\epsilon \\sim N(0,1)}[2 (\\theta+\\epsilon)] $$\n",
    "\n",
    "Next, we convert Expectation to MC estimate to get:\n",
    "\n",
    "$$\\widehat{\\nabla_\\theta J(\\theta)} = \\frac{1}{N} \\sum_{i=1}^{N} 2(\\theta+\\epsilon_i) \\;\\;\\;\\; \\text{...(2)}$$\n",
    "\n",
    "### Experiment\n",
    "\n",
    "Let us say $\\theta=2$\n",
    "\n",
    "1) We generate samples for $x \\sim N(\\theta,1)$ and use these samples in equation (1) to calculate the REINFORCE estimate of $\\nabla_\\theta J(\\theta)$\n",
    "\n",
    "2) We generate samples for $\\epsilon \\sim N(0,1)$ and use these samples in equation (2) to calculate the Reparameterized estimate of $\\nabla_\\theta J(\\theta)$\n",
    "\n",
    "\n",
    "We do so for various values of N and compare the estimate of $\\nabla_\\theta J(\\theta)$ in both the cases. We also calculate the Variance of the estimate in both the cases.\n",
    "\n",
    "\n",
    "**We can see that Variance is lot higher in REINFORCE as compared to Reparameterization - almost higher by one more of magnitude. This is the Apporach of Reparameterization that we will use in Soft Actor Critic.**\n",
    "\n",
    "You can also analytically derive the mean and variance of Equations (1) and (2) as below:\n",
    "\n",
    "### Mean and Variance of Gradient Estimator in Equation (1):  mean=4; variance=87/N\n",
    "\n",
    "$$E[\\widehat{\\nabla_\\theta J(\\theta)}] = E[\\frac{1}{N} \\sum_{i=1}^{N} (x_i-\\theta)  x^2_i]$$\n",
    "$$ = \\frac{1}{N} \\sum_{i=1}^{N} E[(x_i-\\theta)  x^2_i]$$\n",
    "$$ = \\frac{1}{N} \\sum_{i=1}^{N} E[x_i^3]-\\theta E[x^2_i]$$\n",
    "$$ = \\frac{1}{N} \\sum_{i=1}^{N} E[x_i^3]-\\theta E[x^2_i]$$\n",
    "$$ = \\frac{1}{N} \\sum_{i=1}^{N} (\\mu^3+3 \\mu {\\sigma}^2) -\\theta (\\mu^2+\\sigma^2)$$\n",
    "$$ = (\\mu^3+3 \\mu {\\sigma}^2) -\\theta (\\mu^2+\\sigma^2)$$\n",
    "\n",
    "putting $\\theta=\\mu$ and $\\sigma=1$:\n",
    "\n",
    "$$ = (2^3+3.2.1) -2.(2^2+1) = 14-10 = 4$$\n",
    "\n",
    "Again, we calculate the variance of gradient estimate in equation (1)\n",
    "\n",
    "$$Var(\\widehat{\\nabla_\\theta J(\\theta)}) = var(\\frac{1}{N} \\sum_{i=1}^{N} (x_i-\\theta)  x^2_i)$$\n",
    "$$ = \\frac{1}{N^2} \\sum_{i=1}^{N} var(x_i^3-\\theta.x^2_i)$$\n",
    "$$ = \\frac{1}{N^2} \\sum_{i=1}^{N} E[(x_i^3-\\theta.x^2_i)^2] - (E[x_i^3-\\theta.x^2_i])^2 $$\n",
    "$$ = \\frac{1}{N^2} \\sum_{i=1}^{N} E[x_i^6+\\theta^2x_i^4-2.\\theta.x^3_ix^2_i] - (E[x_i^3-\\theta.x^2_i])^2 $$\n",
    "$$ = \\frac{1}{N} \\left[ {\\mu ^{6}+15\\mu ^{4}\\sigma ^{2}+45\\mu ^{2}\\sigma ^{4}+15\\sigma ^{6}}+\\theta^2 ({\\displaystyle \\mu ^{4}+6\\mu ^{2}\\sigma ^{2}+3\\sigma ^{4}})-2.\\theta({\\displaystyle \\mu ^{5}+10\\mu ^{3}\\sigma ^{2}+15\\mu \\sigma ^{4}}) - ({\\displaystyle \\mu ^{3}+3\\mu \\sigma ^{2}}-\\theta.({\\displaystyle \\mu ^{2}+\\sigma ^{2}}))^2 \\right]$$\n",
    "\n",
    "Putting $\\theta=\\mu=2, \\;\\; \\text{and} \\;\\; \\sigma=1$:\n",
    "\n",
    "\n",
    "$$Var(\\widehat{\\nabla_\\theta J(\\theta)}) = \\frac{1}{N}[2^6+15.2^4+45.2^2+15+4(2^4+6.2^2+3)-4(2^5+10.2^3+15.2)-(2^3+3.2-2(2^2+1))^2]$$\n",
    "\n",
    "$$Var(\\widehat{\\nabla_\\theta J(\\theta)}) = \\frac{1}{N}[64+240+180+15+4(16+6.4+3)-4(32+10.8+15.2)-(8+3.2-2(4+1))^2]$$\n",
    "\n",
    "$$Var(\\widehat{\\nabla_\\theta J(\\theta)}) = \\frac{1}{N}[64+240+180+15+4(16+6.4+3)-4(32+10.8+15.2)-(8+3.2-2(4+1))^2]$$\n",
    "\n",
    "$$Var(\\widehat{\\nabla_\\theta J(\\theta)}) = \\frac{87}{N}$$\n",
    "\n",
    "\n",
    "### Mean and Variance of Gradient Estimator in Equation (2):  mean=4; variance=4/N\n",
    "\n",
    "$$E[\\widehat{\\nabla_\\theta J(\\theta)}] = E[\\frac{1}{N} \\sum_{i=1}^{N} 2(\\theta+\\epsilon_i)]$$\n",
    "$$= \\frac{1}{N} \\sum_{i=1}^{N} 2E[(\\theta+\\epsilon_i)]$$\n",
    "$$= \\frac{1}{N} \\sum_{i=1}^{N} 2.\\theta$$\n",
    "$$ E[\\widehat{\\nabla_\\theta J(\\theta)}] = 2.\\theta = 4$$\n",
    "\n",
    "Let us calculate the variance\n",
    "\n",
    "$$Var(\\widehat{\\nabla_\\theta J(\\theta)}) = Var(\\frac{1}{N} \\sum_{i=1}^{N} 2(\\theta+\\epsilon_i))$$\n",
    "$$ = \\frac{1}{N^2} \\sum_{i=1}^{N} Var(2(\\theta+\\epsilon_i))$$\n",
    "$$ = \\frac{1}{N^2} \\sum_{i=1}^{N} Var(2\\epsilon_i)$$\n",
    "$$ = \\frac{1}{N^2} \\sum_{i=1}^{N} 4.Var(\\epsilon_i)$$\n",
    "$$ = \\frac{1}{N^2} \\sum_{i=1}^{N} 4.1$$\n",
    "$$ Var(\\widehat{\\nabla_\\theta J(\\theta)})= \\frac{4}{N}$$\n",
    "\n",
    "### Conclusion from Analytical Solution\n",
    "\n",
    "We can see analytically that Estimate of Gradient in equation (1) and (2) both is equal to **mean=4**. However, Variance in method/equation 1 is $\\frac{87}{N}$ vs that in method 2 being $\\frac{87}{N}$:\n",
    "\n",
    "$$mean_1 = mean_2 = 4$$\n",
    "$$\\frac{var_1}{var_2} = \\frac{87}{4} = 21.75$$\n",
    "\n",
    "We see that REINFORCE/SCORE approach has a higher variance, almost by one order of magnitude higher.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running the Experiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[4.008513  3.9837909 3.9972649 4.0006776 3.9994502]\n",
      "[3.9986959 3.999479  4.0004582 3.9999344 4.0000186]\n",
      "[9.003560e+00 8.494680e-01 8.742704e-02 8.678474e-03 8.554653e-04]\n",
      "[4.1074410e-01 3.9180346e-02 4.0255673e-03 4.0744917e-04 4.0299241e-05]\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "Ns = [10,100,1000,10000,100000]\n",
    "runs=10000\n",
    "\n",
    "theta = 2.0\n",
    "\n",
    "grad_reinforce = lambda x: np.sum((x-theta) * np.square(x)) / x.size\n",
    "grad_reparam = lambda eps: np.sum(2 * (theta+eps)) / eps.size\n",
    "\n",
    "mean_reinforce = np.zeros_like(Ns, dtype=np.float32)\n",
    "var_reinforce = np.zeros_like(Ns, dtype=np.float32)\n",
    "\n",
    "mean_reparam = np.zeros_like(Ns, dtype=np.float32)\n",
    "var_reparam = np.zeros_like(Ns, dtype=np.float32)\n",
    "\n",
    "runs_reinforce = np.zeros(runs)\n",
    "runs_reparam = np.zeros(runs)\n",
    "for i,N in enumerate(Ns):\n",
    "    for r in range(runs):\n",
    "        x = np.random.default_rng().normal(theta, 1, N)\n",
    "        runs_reinforce[r] = grad_reinforce(x)\n",
    "        eps = np.random.default_rng().normal(0, 1, N)\n",
    "        runs_reparam[r] = grad_reparam(eps)\n",
    "    \n",
    "    mean_reinforce[i] = np.mean(runs_reinforce)\n",
    "    var_reinforce[i] = np.var(runs_reinforce)\n",
    "    \n",
    "    mean_reparam[i] = np.mean(runs_reparam)\n",
    "    var_reparam[i] = np.var(runs_reparam)\n",
    "    \n",
    "print (mean_reinforce)\n",
    "print (mean_reparam)\n",
    "print\n",
    "print (var_reinforce)\n",
    "print (var_reparam)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mean_reinforce:\n",
      " [4.008513  3.9837909 3.9972649 4.0006776 3.9994502]\n",
      "\n",
      "mean_reparam:\n",
      " [3.9986959 3.999479  4.0004582 3.9999344 4.0000186]\n",
      "\n",
      "ratio of means:\n",
      " [1.0024551 0.9960774 0.9992018 1.0001858 0.9998579]\n",
      "\n",
      "\n",
      "var_reinforce\n",
      " [9.003560e+00 8.494680e-01 8.742704e-02 8.678474e-03 8.554653e-04]\n",
      "\n",
      "var_reparam\n",
      " [4.1074410e-01 3.9180346e-02 4.0255673e-03 4.0744917e-04 4.0299241e-05]\n",
      "\n",
      "ratio of variances:\n",
      " [21.92012  21.680973 21.717943 21.299526 21.227827]\n"
     ]
    }
   ],
   "source": [
    "print ('mean_reinforce:\\n', mean_reinforce)\n",
    "print ('\\nmean_reparam:\\n', mean_reparam)\n",
    "print ('\\nratio of means:\\n', mean_reinforce/mean_reparam)\n",
    "print ()\n",
    "print ()\n",
    "print ('var_reinforce\\n',var_reinforce)\n",
    "print ('\\nvar_reparam\\n',var_reparam)\n",
    "print ('\\nratio of variances:\\n',var_reinforce/var_reparam)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEICAYAAABWJCMKAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAA3kElEQVR4nO3deXhU5dn48e+dfSF7QhKSYZV9SSAJohbFHRQEVMDaVu3bvpZaW5e21vpai2+X1/Znq7a1UltttZuAyiKiFXdbFQgmhFU2kWyQEMhC9uX5/TEnMIQJmawnk7k/1zVXZs55zpn7HIZzn/V+xBiDUkop3+NndwBKKaXsoQlAKaV8lCYApZTyUZoAlFLKR2kCUEopH6UJQCmlfJQmADUgiMghEbnCev+AiPzJ7pi6QkT+T0TutjsOVyISLCJ7RGSw3bGonqUJQPU6EblJRDaJSLWIlFjv7xAR6Y3vM8b83Bjz9e7OR0SGi4gRkYBztFkmIo0ictLlVe7BvP8iIj9tMywBuAX4Q3dj70nGmHrgWeAHdseiepYmANWrROS7wBPA/wOSgERgKXARENTONP59FmDPWGGMGeTyiu7ifG4DNhhjansutB7zD+BWEQm2OxDVczQBqF4jIlHA/wJ3GGNeNMZUGaccY8yXrD3L1r3hp0Rkg4hUA5eKyLUikiMilSKSLyLL2sz7KyLyuYiUicj/tBm3TET+5vJ5hoh8KCLlIrJNRGa5jHtXRH4iIv8RkSoReUNE4q3R71t/y609+ws6ufwiIo9ZRz0VIpInIpNE5HbgS8B91nxfsSaZA7znMv0sESkQkfuseRSLyAIRuUZE9orIcRF5wKW9n4jcLyIHrPWyUkRiXcavEpEjVizvi8hEl3F/EZEnReRVaz1sEpFRreONMQXACWBGZ9aB6t80AajedAEQDKz1oO3NwM+ACODfQDXO0yHRwLXAN0VkAYCITACeAr4CDAHigFR3MxWRFOBV4KdALPA94CXrdIvrd38VGIzzqOR71vCLrb/R1p79Rx4sh6urrHmMsZZjCVBmjHka+DvwS2u+86z2k4FP28wjCQgBUoCHgD8CXwYygJnAQyIy0mr7HWABcAnO9XICeNJlXq8Bo63l/MSKwdUXgYeBGGA/zn8PV7uBNE8XXvV/mgBUb4oHjhljmloHuOyJ14rIxS5t1xpj/mOMaTHG1Blj3jXGbLc+5wH/xLlhA7gRWG+Med86ivgR0NJODF/GeVplgzWvjUA2cI1Lmz8bY/Zap15WAumdXM7F1jK1vt6xhjfiTGjjADHG7DbGFJ9jPtFAVZthjcDPjDGNwAs41+kT1tHUTmAnMMVq+w3gf4wxBdZ6WQbc2HoNwxjzrDVd67g06yit1cvGmM3Wv9ff3ayHKitGNUBoAlC9qQyId72Iaoy50DpHXsaZv7981wlF5HwReUdESkWkAud1g9ZTM0Nc2xtjqq35uTMMWOS6gQa+ACS7tDni8r4GGOT5IgKw0hgT7fK61IrrbeB3OPfCj4rI0yISeY75nMCZMFyVGWOarfet1waOuoyvdYl3GLDaZTl3A81Aooj4i8gj1umhSuCQNU28y7w6Wg8RQPk54ldeRhOA6k0fAfXAfA/ati1L+w9gHeAwxkQBy4HWu4aKAUdrQxEJw3kayJ184K9tNtDhxphHuhBTpxljfmOMyQAm4jwV9P1zzDvPatNV+cCcNssaYowpxHmaaz5wBRAFDLem6cydWOOBbd2IT/UzmgBUrzHGlOM8p/x7EblRRAZZFyrTgfAOJo8Ajhtj6kRkOs4NWKsXgbki8gURCcJ5obm93/LfgHkicrW1FxxiXVx1e82gjVKcp5ZGdtTQHRHJso5kAnFe06jDuUcOzr34tvPdwOnTXF2xHPiZiAyzvj9BRFqTbwTOZFwGhAE/78yMrWspscDH3YhP9TOaAFSvMsb8ErgXuA8owbnh+wPOe8o/PMekdwD/KyJVOC9+rnSZ507gWziPEopxnjopaOf783Hu+T6Ac4Oej3MvvMPfvjGmBueF0P9Yp1XauwNmiZz5HMBJcT40FYnzou0J4HOcG99HrWmeASZY811jDXseuEZEQjuKrR1P4DxqesNabx8D57vM+3OgENhF5zfkNwPPtd65pQYG0Q5hlOo/ROTnQIkx5nG7Y2klznv/twEXG2NK7I5H9RxNAEop5aP0FJBSSvkoTQBKKeWjNAEopZSParfKYX8UHx9vhg8fbncYSinlVbZu3XrMGJPQdrhXJYDhw4eTnZ1tdxhKKeVVRORzd8P1FJBSSvkoTQBKKeWjNAEopZSP8qprAEqpgamxsZGCggLq6ursDsWrhYSEkJqaSmBgoEftNQEopWxXUFBAREQEw4cPR3qnq+gBzxhDWVkZBQUFjBgxwqNp9BSQUsp2dXV1xMXF6ca/G0SEuLi4Th1FaQJQSvULuvHvvs6uQ59IAB/sK+X37+63OwyllOpXfCIB/HvfMX79xl5Kq7SUuVLKPX9/f9LT05k0aRLz5s2jvLz8nO2XL1/O888/3+F8v/jFLzJlyhQee+yxHoq05/hEAliU6aCpxbA6x22fIUopRWhoKLm5uezYsYPY2FiefPLJc7ZfunQpt9xyyznbHDlyhA8//JC8vDzuuecej+JoamryOObu8okEcN7gQWQMi2HFlny0/wOlVEcuuOACCgsLAThw4ACzZ88mIyODmTNnsmfPHgCWLVvGo486O3ibNWsWP/jBD5g+fTpjxozhgw8+AOCqq66ipKSE9PR0PvjgA3Jzc5kxYwZTpkxh4cKFnDhx4tT0DzzwAJdccglPPPEEW7Zs4cILLyQtLY3p06dTVVVFc3Mz3//+98nKymLKlCn84Q9/6PZy+sxtoEsyHdz3Uh6fHC4nY1iM3eEopdrx8Cs72VVU2aPznDAkkh/Pm+hR2+bmZt566y2+9rWvAXD77bezfPlyRo8ezaZNm7jjjjt4++23z5quqamJzZs3s2HDBh5++GHefPNN1q1bx9y5c8nNzQVgypQp/Pa3v+WSSy7hoYce4uGHH+bxxx8HoLy8nPfee4+GhgbGjRvHihUryMrKorKyktDQUJ555hmioqLYsmUL9fX1XHTRRVx11VUe3/Lpjs8kgGumJLPslZ2s3JKvCUApdZba2lrS09M5dOgQGRkZXHnllZw8eZIPP/yQRYsWnWpXX+/+WuL1118PQEZGBocOHTprfEVFBeXl5VxyySUA3HrrrWfMd8mSJQB8+umnJCcnk5WVBUBkZCQAb7zxBnl5ebz44oun5rdv3z5NAJ4YFBzA3CnJrM8r4qF5EwgP9plFV8qreLqn3tNarwFUVFQwd+5cnnzySW677Taio6NP7cGfS3BwMOC8mNyV8/jh4eGA84Eud7dzGmP47W9/y9VXX93pebfHJ64BtFqc6aC6oZlXtxfbHYpSqp+KioriN7/5DY8++iihoaGMGDGCVatWAc6N8LZt27o835iYmFPXB/7617+eOhpwNW7cOIqKitiyZQsAVVVVNDU1cfXVV/PUU0/R2NgIwN69e6muru5SLK18ajc4Y1gMIxPCWZWdz+JMh93hKKX6qalTp5KWlsYLL7zA3//+d775zW/y05/+lMbGRm666SbS0tK6NN/nnnuOpUuXUlNTw8iRI/nzn/98VpugoCBWrFjBt7/9bWprawkNDeXNN9/k61//OocOHWLatGkYY0hISGDNmjXdWk7xprtiMjMzTXc7hFn+3gEeeW0Pb333EkYlDOqhyJRS3bF7927Gjx9vdxgDgrt1KSJbjTGZbdv61CkggOunpeDvJ6zMzrc7FKWUspXPJYDBESFcOnYwL20tpLG5xe5wlFLKNj6XAACWZDk4drKedz8ttTsUpZSyjU8mgFljE4gfFKyngZRSPs0nE0Cgvx83ZKTw9p4SSqq0ByKllG/yyQQAzmcCmlsMqz8ptDsUpZSyhc8mgFEJg8gcFsOKbC0Qp5TqfDnogcBnEwDA4iwHB0ur+eTwCbtDUUrZrLPloLujubm51+bdGT6dAK6dnEx4kD8rtujFYKXUaZ6Ug77ttttYunQpM2fOZMyYMaxfvx6AQ4cOMXPmTKZNm8a0adP48MMPAXj33Xe59NJLufnmm5k8eTIACxYsICMjg4kTJ/L000+f+v5Bgwbxgx/8gIyMDK644go2b97MrFmzGDlyJOvWreux5fSpUhBthQcHMHfKEF7JK+LH8yZqgTil+oPX7ocj23t2nkmTYc4jHjXtTDnoQ4cO8d5773HgwAEuvfRS9u/fz+DBg9m4cSMhISHs27ePL37xi7RWMNi8eTM7duw4VcHz2WefJTY2ltraWrKysrjhhhuIi4ujurqaWbNm8Ytf/IKFCxfy4IMPsnHjRnbt2sWtt97Kdddd1yOrxee3eIuzUlmRnc+recUsztL6QEr5qq6Ug168eDF+fn6MHj2akSNHsmfPHkaMGMGdd95Jbm4u/v7+7N2791T76dOnn1G++Te/+Q2rV68GID8/n3379hEXF0dQUBCzZ88GYPLkyQQHBxMYGMjkyZPdlpruKp9PANOGxjAqIZwV2fmaAJTqDzzcU+9pXSkH3bZss4jw2GOPkZiYyLZt22hpaSEkJOTU+NaSz+A8JfTmm2/y0UcfERYWxqxZs6irc96WHhgYeGrefn5+p0pN+/n59WiXkT59DQCc/2CLMx1s/fwE+0tO2h2OUspmnSkHvWrVKlpaWjhw4AAHDx5k7NixVFRUkJycjJ+fH3/961/bveBbUVFBTEwMYWFh7Nmzh48//rhPls+VzycAgOunpeLvJ6zSJ4OVUpxdDvqZZ54hLS2NiRMnsnbt2lPtxo4dyyWXXMKcOXNYvnw5ISEh3HHHHTz33HPMmDGDvXv3nrHX72r27Nk0NTUxZcoUfvSjHzFjxoy+WrxTfK4cdHv++/lscg6X89EPLyPQX/OiUn3JG8tB33bbbcydO5cbb7zR7lDOoOWgu2BJprNA3Dt7SuwORSml+oTHCUBE/EUkR0TWuxknIvIbEdkvInkiMs1l3GwR+dQad7/L8GUiUigiudbrmu4vTtfNGptAQkQwK7ML7AxDKeUl/vKXv/S7vf/O6swRwF3A7nbGzQFGW6/bgafAmTSAJ63xE4AvisgEl+keM8akW68NnQ2+JwX4+3HDtFTe+bSEkkotEKdUX/Om09H9VWfXoUcJQERSgWuBP7XTZD7wvHH6GIgWkWRgOrDfGHPQGNMAvGC17ZcWZ6bS3GJ4OUcLxCnVl0JCQigrK9Mk0A3GGMrKys647bQjnj4H8DhwHxDRzvgUwPUWmgJrmLvh57t8vlNEbgGyge8aY84qyiMit+M8qmDo0KEehts1IxMGkTU8hpVb8vnGxSPPusdXKdU7UlNTKSgooLRUO2nqjpCQEFJTUz1u32ECEJG5QIkxZquIzGqvmZth5hzDwXma6CfW558AvwL+66zGxjwNPA3Ou4A6ire7Fmc6+P6LeWz9/ASZw2N7++uUUjgffHJ9Qlb1DU9OAV0EXCcih3CewrlMRP7Wpk0B4PoYbSpQdI7hGGOOGmOajTEtwB9xni6y3TVaIE4p5SM6TADGmB8aY1KNMcOBm4C3jTFfbtNsHXCLdTfQDKDCGFMMbAFGi8gIEQmypl8HYF0jaLUQ2NH9xem+8OAA5qUN4dXtxZys77lHrpVSqr/p8nMAIrJURJZaHzcAB4H9OPfm7wAwxjQBdwL/wnkH0UpjzE5rml+KyHYRyQMuBe7paiw9bVGmg5qGZl7NK7I7FKWU6jX6JLAbxhiufOx9IkMCePmOi3r9+5RSqjfpk8Cd4CwQl8onh8vZX1JldzhKKdUrNAG0Y+HUVAL8RJ8MVkoNWJoA2pEQEcxl4wbz8icFNDa32B2OUkr1OE0A57Aky8Gxkw28rQXilFIDkCaAc7hkTAKDI4K1nwCl1ICkCeAcAvz9uCEjlXc+LdUCcUqpAUcTQAcWZzpobjG89IkWiFNKDSyaADowIj6c6cNjWZWdr5UKlVIDiiYADyzOcnDwWDVbDp1VrFQppbyWJgAPXDM5iUHBAazUi8FKqQFEE4AHwoICmJeWzKt5xVTVNdodjlJK9QhNAB5alOmgtrGZV/OK7Q5FKaV6hCYAD011RDN68CBW6GkgpdQAoQnAQ84CcQ5yDpez76gWiFNKeT9NAJ2wcFqKVSBOjwKUUt5PE0AnxA8K5vLxg3n5k0ItEKeU8nqaADppSZaDsuoG3tqtBeKUUt5NE0AnXTw6gcRILRCnlPJ+mgA6KcDfjxumpfLOpyUc1QJxSikvpgmgCxZnOmgx8OJW7S1MKeW9NAF0wfD4cKaP0AJxSinvpgmgi5ZkOjhUVsPmz47bHYpSSnWJJoAumnOqQJyeBlJKeSdNAF3kLBA3hA3btUCcUso7aQLohsWZqdQ2NrNeC8QppbyQJoBuSHdEMyZxECu26DMBSinvowmgG1oLxOXml7NXC8QppbyMJoBuWjjVKhCnRwFKKS8TYHcA3i5uUDBXjE9kdU4h980eR1CA5lSl3DHGsK2ggjU5hXx8sIz4QcE4YkNJjQnDERuGIyYUR2wYceFBiIjd4foETQA9YEmWg9d3HuHtPUeZPSnZ7nBUX2uqh7oKCAqHwDDQjdcZPi+rZk1OEWtyC/nsWDVBAX6cPyKWyrom3th5lLLqhjPahwX5kxoTisNKDKlWYnB+DiUiJNCmJRl4NAH0gIvHJJAUGcKKLfmaAAaSlmaoPgZVRVB1BKqKobLY+bf1c1Ux1JSdnkb8IDgCgiPb/HV9uRkWEnXm58Bw8PPeo8myk/W8ur2Y1TmF5BwuRwRmjIjjm5eMYvbkJCJdNuLV9U0UnKgl/3gN+SdqyD9ea/2tYdNnxzlZ33TGvKPDAk8lA0dMGKkuRw8p0aGEBPr39eJ6LY8TgIj4A9lAoTFmbptxAjwBXAPUALcZYz6xxs22xvkDfzLGPGINjwVWAMOBQ8BiY8yJbi6PLfz9hBsyUnjq3QMcqagjKSrE7pDUuRgDdeXOjXhl0Zkbc9cN/cmjYJrPnFb8IHwwRCRB9FBwTIeIZAiNgYZqqK9yeVU6/9aUwYlDp4c3VnsQpLhJHK4JxMPEEhwBfn2zQaxtaObN3UdZk1PIe3tLaWoxjEuK4P4547gubQhDokPdThceHMDYpAjGJkWcNc4YQ3lN46nEcPhUkqhhd3EVb+4qoaFN3xyJkcGnjh4cMaFWgnAmjOSoUPz99AitVWeOAO4CdgORbsbNAUZbr/OBp4DzraTxJHAlUABsEZF1xphdwP3AW8aYR0TkfuvzD7q8JDZblOHgyXcO8NInBXzr0vPsDsd3NdScvYd+1ob+CDTVnj1taIxzYx6RBAnjnX8jkiByiPU+2bnx9+/mgXNzEzRUtUkWLgnD9VVXeXp4XSVUFJ4e1+DhnWdBg86RSNx9bieR+J996qW5xfDRgTJW5xTy+o5iqhuaSYoM4WszR7AgPYXxye42F54TEWLCg4gJD2JKavRZ41taDEer6pxHDW2OIDYdLGNNZR2u5boC/ISUU6eX9PqDR79kEUkFrgV+Btzrpsl84HnjrIz2sYhEi0gyzr37/caYg9Z8XrDa7rL+zrKmfw54Fy9OAMPjwzl/RCwrs/O5Y9Yon/oR9YnmRjhZco6NuvWqqzh72sAwa8OeDCkZEJl8ekMfMeT0hj7Q/R5qj/MPcCab0JjuzaelBRpOtkkgbpJI2+RSVwlVR88chwdFDQNCITgCExxBnX84R+oC+fykP2VNwWT4hXNtYgKjUpNxJCfiF1IGlYegPvLsBBMQ5PkyGuM8FWdaXF6nP/u1tJDs30JyfAvT4/zBhIEJARMFpoWGxiZKK2s4UlHDkfJajlZUU1JRQkllLQVFteyprUcw+NGCP4aQAEiMDCJxUBCJgwKJHxTA4EFBxIcHkhAeQGignI6jxX1MznHNztjbHd/S7jK1O+30b8DgcV39tbj/J/Ww3ePAfcDZx2hOKYDrfZAF1jB3w8+33icaY4oBjDHFIjLY3YxF5HbgdoChQ4d6GK49lmQ5uHflNjZ9dpwZI+PsDsc7tLRA7XE359fbnHevLuWsjZRfAAxKcm7Q40fDiItPb+hd99yDIwfmhVk/PwiJdL66w5j2T1+5JI6qiuMcLj5KSWkppr6KCDnJqJBGpofWE9pSjRyrgtJmyOng+/yDISjM+u4ONobdFMTpDZFbwW6GVVuvo93+eg+J89Sin7/zr/iBtL53GTdhQd8nABGZC5QYY7aKyKz2mrkZZs4x3GPGmKeBpwEyMzP7de3lOZOS+fHanazMztcEANaeZps99DP23K1xLW5qKYUnnN5DT053btTb7rmHxXn1hdJ+QwSCBzlfnHkTQ0VNIxt2FLN6T+GpyrdZw2OYn57CtZOTiQl32Zs3Bhprz31Kq/UopaHaZWPn5nXGxrAT4zo93v/0RrbNOINQ1WA4WtXAkcoGjlQ1UFzZ+qqnuKKBuhZosY4hmo0fcRHBJEWFkxwTRnJ0OENinK+UmDCSosPx93f33fbtnHhyBHARcJ2IXAOEAJEi8jdjzJdd2hQADpfPqUARzgTsbjjAURFJtvb+kwGv72Q3NMifeelDePmTApZdN/GMOx0GlKb6sy+Yujvv3nDy7GmDI09vxIdd6P50zKDEzp0mUD2qvqmZd/aUsDqnkHf2lNLQ3MKohHC+d9UY5qen4IgNcz+hiHPPPigMIhL7NuheIDgveEbivLjZVkuLoaSq/tRFade7l3bn11K84xgt5tip9gF+wpDo0FN3L7W9xTV+UN9ff+gwARhjfgj8EMA6Avhem40/wDrgTusc//lAhbVhLwVGi8gIoBC4CbjZZZpbgUesv2u7vTT9wOJMB//YdJj124q5+fz+fcrqDMY4N9jVx5x3rZw63+5yOqZ1Q1/rpg8E/+DTp12SJsPoq84+HTMo0drLVP1NS4th86HjrM0t5NW8YirrmogfFMxXLhjGgvQUJqVE6nWtNvz8hKSoEJKiQsgaHnvW+IamFooras9IDPnW7a5v7j7KsZNnPv8QGujvkhBCrQRh3e4aG9YrO5Rdvp1BRJYCGGOWAxtw3gK6H+dtoF+1xjWJyJ3Av3DeBvqsMWanNYtHgJUi8jXgMLCoq7H0J2mpUYxNjGBFdr69CcAY5wXR6mNQc6zN3zL3n5vrz56P+Dk33BFJEDMchs5wOR2TdHojHxozMM+zD3B7j1axOqeQtTmFFFXUERbkz+yJSSyYmsKFo+II8NdTbF0VFODHsLhwhsWFux1f0+Dy/MPxGg67JIrNbp5/+OMtmVw5oWePrMSbujTMzMw02dnZdofRoT99cJCfvrqbf919sdt7m7ukpQVqT3i+Ma8pc39uHZy3BYbFQXg8hMVbf63P4Qmnh0Ukw6DBfXYfueobRyvrWJdbxOqcQnYVV+LvJ1w8Op4FU1O4ckIiYUH6fKjd2j7/kH+ihnlpQ0hp51mKjojIVmNMZtvh+i/dC66flsovXt/Dyux8fjR3gvtGzU3OUykebcyPQc3xsx9KahUcBeFxzg139FBImeayYY8/Pa71c6A+qOZrquoaeX3HEdbkFvLhgTKMgTRHNMvmTWBu2hDiB7m7HUbZpaPnH3qKJoCe0tTg3GDXHCO2+hg/dOzg2NY3aA6Jw7+2zNqot/4thdpy2r0hKjTm9J54/HkQNsP93nqY9V4vmCo3GppaeH9vKWtyC9m46yj1TS0MiwvjO5eNZsHUFEbEuz81oXyHJoD2NNa1Od1Sdvbn6tLT7+vPfADpv6y/5t9+Z+6BD57Q/sY8PB5CY7v/pKnyWcYYPjlczpqcQtbnFXGippHY8CCWZDlYMDWFqY5ovZirTvGdLU1D9Tk25m7Oq7u7hRGcDx+5bryHTHW7MW8OjWPOn/aQkpzEn/9rRt8uq/I5B0pPsjankDW5RRw+XkNwgB9XTUxi4dQhzBydQKBezFVu+EYCWH8PZD/rfpx/8Jkb79hR7V8UDYtzVm30YA/KH7gqU/j9u/sprqglOaqPygwon1FaVc/6vCLW5BSyraACEbhoVDzfuXw0V09M1LLJqkO+kQDGzIHoYe5PvQQN6rXbFxdlpvK7d/bz0tYC7rzM3aMkSnVOTYOzhv6a3EI+2HeM5hbDxCGRPHjteOalDSExUi/wK8/5SAK4yvnqY8PiwpkxMpaV2QXcMes8/LQMreqCpuYW/nOgjDU5hfxr5xFqGppJiQ7lGxePZMHUFMYk9tCtxsrn+EYCsNGSLAf3rHAWiLtglNYHUp4xxrC9sILVOYW8sq2YYyfriQwJYH56CgunppA5LEZ3KFS3aQLoZbMnJvNQ8E5WZedrAlAdOlxWw9rcQlbnFnKwtJogfz8uHz+Y+ekpXDougeAAfShP9RxNAL0sNMif69KH8NInBSybP4ALxKkuO1HdwPrtxazJKWTr585O8c4fEcvtM0cyZ1IyUWH6m1G9QxNAH1ic6eDvmw7zyrYivnT+MLvDUf1AXWMzb+12Vtx8b28Jjc2GMYmDuG/2WOanp3T5kX+lOkMTQB+YkhrFuKQIVm7J1wTgw5pbDJsOtnafeISq+iYSI4P56kWt3SdG6ENaqk9pAugDIsKiTAc/Wb+LPUcqGZfUzR6clFfZXVzJmpxC1uYWcaSyjkHBAcyZ5Ky4OWNknHZSrmyjCaCPLJyawiOv7WbllgIemtdOgTg1YBSV17I21/mQ1qdHqwjwE2aNTeDBueO5YnwiIYF6MVfZTxNAH4kND+KqCUmszingB3PG6t0cA1BFbSOv7yhmdU4hmz47jjGQMSyGnyyYxLWTk4kN16J9qn/RBNCHFmWm8ur2Yt7aXcI1k5M7nkB5hcLyWn726i7e3F1CQ1MLI+PDueeKMcxPH9JuZyBK9QeaAPrQzNEJJEeFsGJLviaAAaKxuYVv/f0T9h2t4ubpQ1k4NYUpqVF6MVd5BU0AfcjfT7gxw1kfqKi8liF6q5/X+93b+8nNL+d3N09l7pQhdoejVKdojdg+tijDgTHw0tYCu0NR3bT18xP89u19XD8tRTf+yitpAuhjQ+PCuGBkHKu2FtDS4j39Masznaxv4p4VuaTEhPLwdRPtDkepLtEEYIMlWQ4OH6/h48/K7A5FddGydTspOFHDY4vTte6+8lqaAGwwe1ISESEBrMrW00DeaMP2Yl7cWsC3Lj2PzOGxdoejVJdpArBBSKA/89OHsGF7MRW1jXaHozrhSEUdP3x5O2mOaL5zuXbyo7ybJgCbLM50UN/UwrptRXaHojzU0mL47qpcGptbeHxJuvazq7ye/oJtMjnFWSBuVXa+3aEoDz37n8/4z/4yHpo7gRHx+oCX8n6aAGwiIizJcpBXUMHu4kq7w1Ed2FVUyS9f/5SrJiSyJMthdzhK9QhNADZakJ5CkL8fK/UooF+ra2zm7hU5RIUF8sgNU/QpXzVgaAKwUUx4EFdOTGR1TiH1Tc12h6Pa8YvX97D36EkeXZSmBd3UgKIJwGaLMx2U1zTy5q4Su0NRbry/t5Q//+cQt104nEvGJNgdjlI9ShOAzb5wXjxDokJYoaeB+p3j1Q18d9U2xiQO4v454+wOR6kepwnAZq0F4j7YV0pRea3d4SiLMYb7X8qjoqaRx5dM1Q5c1IDUYQIQkRAR2Swi20Rkp4g87KZNjIisFpE8q+0kl3F3icgOa9q7XYYvE5FCEcm1Xtf02FJ5mUWZzgJxL2qBuH5jxZZ83th1lO9fPZYJQ7QLTzUweXIEUA9cZoxJA9KB2SIyo02bB4BcY8wU4BbgCQArEfw3MB1IA+aKiOvjk48ZY9Kt14buLYr3csSGceGoOFZtzdcCcf3AZ8eqefiVXVx0Xhxf+8IIu8NRqtd0mACM00nrY6D1aruVmgC8ZbXfAwwXkURgPPCxMabGGNMEvAcs7KngB5IlWQ7yj9fy8UEtEGenxuYW7l6RS1CAH48uSsNPO2xXA5hH1wBExF9EcoESYKMxZlObJtuA662204FhQCqwA7hYROJEJAy4BnB9iuZO67TRsyIS08533y4i2SKSXVpa2pll8ypXT3QWiNOLwfb67Vv72JZfzv9dP5nkKO2wRw1sHiUAY0yzMSYd50Z9uus5fssjQIyVJL4N5ABNxpjdwC+AjcDrOBNFkzXNU8AonKeVioFftfPdTxtjMo0xmQkJA/c2vJBAfxakp/DajiNU1GiBODtkHzrO797Zz40Zqdplp/IJnboLyBhTDrwLzG4zvNIY81UrSdwCJACfWeOeMcZMM8ZcDBwH9lnDj1qJpQX4I87rBD5tcaaDhqYW1m0rtDsUn1NV18jdVgcvP543we5wlOoTntwFlCAi0db7UOAKYE+bNtEi0vqI5NeB940xlda4wdbfoThPE/3T+uy6i7UQ5+kinzYpJZLxyZGs1H4C+tyydbsoKq/l8SXawYvyHZ50Cp8MPCci/jgTxkpjzHoRWQpgjFmO82Lv8yLSDOwCvuYy/UsiEgc0At8yxpywhv9SRNJxXlA+BHyjB5bHq4kISzJTWfbKLnYVVerth33k1bxiXvqkgO9cPpqMYdrBi/IdYoz33HaYmZlpsrOz7Q6jV52obuD8n7/FzecPZZn2Ndvriitqmf34B4yID+fFpRcQoDX+1QAkIluNMZlth+uvvZ+JCQ/iqomJrMnVAnG9raXF8N2V20518KIbf+Vr9BffD7UWiNu466jdoQxof/r3QT48UMayeRMZrh28KB+kCaAfuui8eFKiQ1mxRZ8J6C27iir5f//6lKsnJrIoM9XucJSyhSaAfsjfT7ghI5V/7z9GoRaI63F1jc3c9UIOMWFBPHK9dvCifJcmgH5qUUaqs0Cc3hLa4x55bQ/7Sk7yq8VpxGgHL8qHaQLopxyxYVx0nhaI62nvflrCXz48xH9dNIKZowfuk+VKeUITQD+2ONNBwYlaPtICcT2i7GQ931uVx9jECO6bPdbucJSynSaAfuzqiUlEhgToxeAeYIzh/pe3U1nbyOM3pWsHL0qhCaBfCwn0Z8HUFF7fqQXiuuuFLfls3HWU+2aPZXyyPmGtFGgC6PdaC8St1QJxXXaw9CT/+8ouvnBePP91kXbwolQrTQD93KSUKCYkR7JS+wnoksbmFu5ZkUtwoB+/WqwdvCjlShOAF1iS5WBHYSU7iyrsDsXrPPHmPrYVVPDI9ZNJjAyxOxyl+hVNAF5gfvoQggL8WKXPBHTKlkPH+f27+1mUkcrsSdrBi1JtaQLwAtFhQVw9MYnVOYXUNWqBOE9U1jVyz4pcHLFh/FirqirlliYAL7E4M5WKWi0Q56lla3dSXFHHY0vSGRTsSbcXSvkeTQBe4qJRzgJxejG4Y69sK+LlnEK+fdl5TBsaY3c4SvVbmgC8hJ+fcKNVIK7gRI3d4fRbReW1/M/q7UwdGs2dl55ndzhK9WuaALxIa9niF7fqxWB3mlsM967MpbnFaAcvSnlA/4d4kdSYMC4aFc+q7AItEOfGnz44yMcHj/Pj6yYyLE47eFGqI5oAvMziLAeF5bV8eEALxLnaUVjBo298ypxJSSzK0A5elPKEJgAvc9WERKJCA1mhF4NPqW1o5u4VucSGB/HzhZO1gxelPKQJwMuEBPqzIH0I/9p5hPKaBrvD6Rf+77Xd7C85ya8WpWsHL0p1giYAL7Q4yyoQl1tkdyi2e2dPCc9/9Dlf/8IIvjA63u5wlPIqmgC80MQhUUwcogXijp2s5/svbmNcUgTfu1o7eFGqszQBeKklWQ52FlWyo9A3C8QZY7j/pTwq65p44qap2sGLUl2gCcBLzU9LsQrE+eZRwD82H+bN3SXcP3scY5Mi7A5HKa+kCcBLRYUFMntiEmtyi3yuQNyB0pP8ZP0uZo6O57YLh9sdjlJeSxOAF1uc6aCitpF/7Txidyh9pqGphbtfyCU00J9HF2kHL0p1hyYAL3bhqDhSokN9qp+AJ97ay/bCCv7v+inawYtS3aQJwIv5+QmLMp0F4vKPD/wCcZs/O87v3z3AkkwHsycl2R2OUl5PE4CXuzEjFZGBXyCutYOXYbFhPDRvgt3hKDUgdJgARCRERDaLyDYR2SkiD7tpEyMiq0Ukz2o7yWXcXSKyw5r2bpfhsSKyUUT2WX+1cHsXpMaE8YXz4nlxawHNA7hA3ENrdnCk0tnBS7h28KJUj/DkCKAeuMwYkwakA7NFZEabNg8AucaYKcAtwBMAViL4b2A6kAbMFZHR1jT3A28ZY0YDb1mfVRcszmwtEHfM7lB6xdrcQtbkFnHX5aOZqh28KNVjOkwAxumk9THQerXd1ZyAcyOOMWYPMFxEEoHxwMfGmBpjTBPwHrDQmmY+8Jz1/jlgQTeWw6dd2VogbsvAeyagsLyWB9fsIGNYDHfMGmV3OEoNKB5dAxARfxHJBUqAjcaYTW2abAOut9pOB4YBqcAO4GIRiRORMOAawGFNk2iMKQaw/g5u57tvF5FsEckuLS3t1ML5ipBAfxZOTeGNnUcHVIG45hbDvStyMQYeW6wdvCjV0zz6H2WMaTbGpOPcqE93PcdveQSIsZLEt4EcoMkYsxv4BbAReB1nomjqTIDGmKeNMZnGmMyEhITOTOpTFmc6aGhuYU1Ood2h9Jin3z/Ips+Os+y6iQyNC7M7HKUGnE7tUhljyoF3gdlthlcaY75qJYlbgATgM2vcM8aYacaYi4HjwD5rsqMikgxg/S3p+mKoCUMimZQSycoB8kzAjsIKfr3xU66dnMwN01LsDkepAcmTu4ASRCTaeh8KXAHsadMmWkRaC7F/HXjfGFNpjRts/R2K8zTRP61264Bbrfe3Amu7tSSKJZkOdhV7f4G42oZmvvNCDnHhwfxs4STt4EWpXuLJEUAy8I6I5AFbcF4DWC8iS0VkqdVmPLBTRPYAc4C7XKZ/SUR2Aa8A3zLGnLCGPwJcKSL7gCutz6obrrMKxHn7xeCfb9jNwdJqfrU4jegw7eBFqd7S4Q3Vxpg8YKqb4ctd3n8EjG7bxho3s53hZcDlHkeqOhQVFsicSUmszS3kf64d75Ulkt/ec5S/fvw5/z1zBBedpx28KNWb9LaKAWZxpoPKuiavLBB37GQ9972Yx/jkSO3gRak+oAlggLlgZBypMaFe11uYMYb7Xsyjqq6JJ25KJzjA+45elPI2mgAGGD8/YVGGg//sL/OqAnF/23SYt/eU8MM54xiTqB28KNUXNAEMQDdmOgvErfKSAnH7S07ys1d3cfGYBG7VDl6U6jOaAAaglOhQZ4G47Px+XyCuoamFu1fkEBYUwKM3TtFbPpXqQ5oABqglWQ6KKur4z/7+XSDusTf3sqOwkkeun8xg7eBFqT6lCWCAunJCItFhgazoxxeDPz5YxvL3DvDF6Q6umqgdvCjV1zQBDFDBAf4sSE9h486jnKjufwXiKmobuXdFLsPjwvnRXO3gRSk7aAIYwE4ViMvtfwXiHlq7g6NV9Ty+JJ2wIO3gRSk7aAIYwCYMiWRyShQrtuRjTP+5GLw2t5C1uUXcfflo0hzRdoejlM/SBDDALc5ysOdIFTsKK+0OBYCCEzU8uHoHmcNiuOPS8+wORymfpglggLsubQjBAX6syD5sdyhWBy/bMMBjS9Lx99NbPpWykyaAAS4qtLVAXBF1jc22xrL8vQNsPnSc/50/EUesdvCilN00AfiAxZkOquqaeH2HfQXi8grKeWzjXuZOSWbhVO3gRan+QBOAD5gxMg5HrH0F4moamrj7hVwSIoL52YLJ+rSvUv2EJgAf0Fog7sMDZRwu6/sCcT97dTeflTk7eIkKC+zz71dKuacJwEfcmOEsEPfi1r49Cnhz11H+vukwt88cyYWjtIMXpfoTTQA+Ykh0KDNHJ7Bqa0GfFYgrrarnBy/lMSE5knuvGtMn36mU8pwmAB+yJNNBcUUd/+6DAnHODl62cbJeO3hRqr/SBOBDrpgwmJiwQFb2Qafxf/v4c975tJQHrhnPaO3gRal+SROADwkO8GfB1BTe2HWE471YIG5/SRU/fXU3s8YmcMsFw3rte5RS3aMJwMcsyXLQ2GxYk9M7BeIamlq464VcwoMD+KV28KJUv6YJwMeMS4pkSmoUK7N7p0DcrzZ+ys6iSn55wxQGR2gHL0r1Z5oAfNDiTGeBuO2FFT06348OlPH0+we5+fyhXDEhsUfnrZTqeZoAfNC81gJxPXgxuKKmkXtX5jIiLpwHrx3fY/NVSvUeTQA+KCo0kGsmJ7Mut4jahu4XiDPG8ODaHZRW1fP4TdrBi1LeQhOAj1qUmUpVfROv7yzu9rzW5BbyyrYi7rlyDFNSo7sfnFKqT2gC8FEzRsQxNDaMlVsKujWf/OM1PLRmJ9OHx7L0klE9FJ1Sqi9oAvBRzgJxqXx0sIzPy6q7NI/mFsO9K3MB+PWSNO3gRSkvownAh92Y2VogrmtHAcvfO8CWQyf4yYJJpMZoBy9KeRtNAD4sOSqUi0cn8GIXCsS1dvAyL20I89OH9FKESqne1GECEJEQEdksIttEZKeIPOymTYyIrBaRPKvtJJdx91jT7RCRf4pIiDV8mYgUikiu9bqmZxdNeWJJlrNA3Pv7Sj2eprWDl8ERwfx0wSR92lcpL+XJEUA9cJkxJg1IB2aLyIw2bR4Aco0xU4BbgCcARCQF+A6QaYyZBPgDN7lM95gxJt16bejeoqiuuGJ8IrHhQazqRG9hP1nv7ODl10vSiQrVDl6U8lYdJgDjdNL6GGi92p4vmAC8ZbXfAwwXkdZHQQOAUBEJAMKAop4IXPWMoAA/FqSnsHHXUcpO1nfY/o2dR/jn5sN84+JRzBgZ1wcRKqV6i0fXAETEX0RygRJgozFmU5sm24DrrbbTgWFAqjGmEHgUOAwUAxXGmDdcprvTOm30rIjEtPPdt4tItohkl5Z6fppCee5Ugbjcc+fmkqo67n95OxOHRHLvldrBi1LezqMEYIxpNsakA6nAdNdz/JZHgBgrSXwbyAGarI36fGAEMAQIF5EvW9M8BYzCeVqpGPhVO9/9tDEm0xiTmZCQ0IlFU54amxRBWmoUK7e0XyDOGMP3V+VRbXXwEhSg9w8o5e069b/YGFMOvAvMbjO80hjzVStJ3AIkAJ8BVwCfGWNKjTGNwMvAhdY0R63E0gL8EZjevUVR3bE4y8GnR6vIK3BfIO75jz7nvb2lPHjteM4brB28KDUQeHIXUIKIRFvvQ3Fu1Pe0aRMtIkHWx68D7xtjKnGe+pkhImHivFXkcmC3NU2yyywWAju6uSyqG+alDSEk0I8Vbi4G7ztaxc837OaycYP58gzt4EWpgcKTI4Bk4B0RyQO24LwGsF5ElorIUqvNeGCniOwB5gB3AVjXCl4EPgG2W9/3tDXNL0VkuzXfS4F7emqhVOdFhgRyzaRkXmlTIK6+qZnvvJDLoOAAfnGDdvCi1EDSYdlGY0weMNXN8OUu7z8CRrcz/Y+BH7sZ/pVORap63aJMBy/nFPLajmKun5YKwK/f2Mvu4kqeuTWThIhgmyNUSvUkvZKnTpkxMpZhcWGstE4DfXjgGE9/cJAvnT+Uy8drBy9KDTSaANQpIs4CcR8fPE5eQTnfXbmNEfHhPHjtBLtDU0r1Ak0A6gw3ZKTiJ/ClP22itKqeJ5ZMJTTI3+6wlFK9QBOAOkNyVCgXj0mgqq6Je68aw+TUKLtDUkr1Eu27T53lgWvGM21oDN+4WDt4UWog0wSgzjImMYIxifqwl1IDnZ4CUkopH6UJQCmlfJQmAKWU8lGaAJRSykdpAlBKKR+lCUAppXyUJgCllPJRmgCUUspHSXtdAPZHIlIKfN7FyeOBYz0YTk/RuDpH4+ocjatz+mtc0L3YhhljzupT16sSQHeISLYxJtPuONrSuDpH4+ocjatz+mtc0Dux6SkgpZTyUZoAlFLKR/lSAni64ya20Lg6R+PqHI2rc/prXNALsfnMNQCllFJn8qUjAKWUUi40ASillI8acAlARGaLyKcisl9E7nczXkTkN9b4PBGZ1k/imiUiFSKSa70e6oOYnhWREhHZ0c54u9ZVR3H1+bqyvtchIu+IyG4R2Skid7lp0+frzMO47Ph9hYjIZhHZZsX1sJs2dqwvT+Ky5Tdmfbe/iOSIyHo343p2fRljBswL8AcOACOBIGAbMKFNm2uA1wABZgCb+klcs4D1fby+LgamATvaGd/n68rDuPp8XVnfmwxMs95HAHv7ye/Lk7js+H0JMMh6HwhsAmb0g/XlSVy2/Mas774X+Ie77+/p9TXQjgCmA/uNMQeNMQ3AC8D8Nm3mA88bp4+BaBFJ7gdx9TljzPvA8XM0sWNdeRKXLYwxxcaYT6z3VcBuIKVNsz5fZx7G1eesdXDS+hhovdredWLH+vIkLluISCpwLfCndpr06PoaaAkgBch3+VzA2f8RPGljR1wAF1iHpa+JyMRejskTdqwrT9m6rkRkODAV596jK1vX2TniAhvWmXU6IxcoATYaY/rF+vIgLrDnN/Y4cB/Q0s74Hl1fAy0BiJthbTO7J216miff+QnOeh1pwG+BNb0ckyfsWFeesHVdicgg4CXgbmNMZdvRbibpk3XWQVy2rDNjTLMxJh1IBaaLyKQ2TWxZXx7E1efrS0TmAiXGmK3nauZmWJfX10BLAAWAw+VzKlDUhTZ9HpcxprL1sNQYswEIFJH4Xo6rI3asqw7Zua5EJBDnRvbvxpiX3TSxZZ11FJfdvy9jTDnwLjC7zShbf2PtxWXT+roIuE5EDuE8TXyZiPytTZseXV8DLQFsAUaLyAgRCQJuAta1abMOuMW6mj4DqDDGFNsdl4gkiYhY76fj/Lcp6+W4OmLHuuqQXevK+s5ngN3GmF+306zP15kncdmxzkQkQUSirfehwBXAnjbN7FhfHcZlx/oyxvzQGJNqjBmOcxvxtjHmy22a9ej6Cuh6uP2PMaZJRO4E/oXzzptnjTE7RWSpNX45sAHnlfT9QA3w1X4S143AN0WkCagFbjLWZf/eIiL/xHm3Q7yIFAA/xnlBzLZ15WFcfb6uLBcBXwG2W+ePAR4AhrrEZsc68yQuO9ZZMvCciPjj3ICuNMast/v/o4dx2fUbO0tvri8tBaGUUj5qoJ0CUkop5SFNAEop5aM0ASillI/SBKCUUj5KE4BSSvkoTQBKKeWjNAEopZSP+v/JfnJPULkRWwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "fig = plt.figure()\n",
    "ax = plt.subplot(111)\n",
    "\n",
    "ax.plot(mean_reinforce, label='Reinforce')\n",
    "ax.plot(mean_reparam, label='Reparam')\n",
    "\n",
    "plt.title('Gradient Est(mean)')\n",
    "ax.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAEICAYAAAB25L6yAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAoc0lEQVR4nO3deXxc5X3v8c9vtMvWZlm2ZUvWGDAYjPcFWVxqEyB2wIGEQIAQI/fVlFCa3LRNCW3SkND25iYtLQmUC6EhxRgHCCQhlJKEJTGQeN8wi43BjmzJCxa2Jcu2ZG3P/eMciZGsZWRrdGak7/v1GjwzZ5nfPC/xnWee88w55pxDRETiVyjoAkREpGcKahGROKegFhGJcwpqEZE4p6AWEYlzCmoRkTinoJZ+Y2YVZna5f//rZvajoGs6HWb2f83sr2K074fM7Jux2HeUrz/VzFYF9fpyehTUQ4SZ3Whma83suJkd9O/fbmYWi9dzzn3HOfeFM92PmYXNzJlZcg/rfNvMmszsWMStJop9P2pm/9zpuQLgFuCHZ1p7V5xztznn/ikW+47y9bcCNWb2yaBqkL5TUA8BZvZV4AfAvwJjgNHAbcDFQGo32yQNWIH94ynn3PCIW+5p7mcp8IJzrr7/SvPEUZuuAL4YdBESPQX1IGdmOcA/Arc7555xztU5z2bn3M3OuZP+eo+a2YNm9oKZHQcuNbOrzGyzmR01s0oz+3anfS8xs91mdsjMvtFp2bfN7PGIx6VmtsrMaszsDTNbELFspZn9k5n9wczqzOxFMxvpL37N/7fG7ynP6+P7NzO71/8WUWtmW83sQjO7FbgZ+Jq/3//2N/kE8GrE9tvMbHHE42Qz+9DMZvqPnzazA/6+XzOzyRHrdtWm7b14M8szs+fNrNrMjvj3i6JsF8zsf0W0aaWZLfWfTzOze8xsj5l94A+3ZEQ0y0rgMjNL60tbSnAU1IPfPCAN+GUU634O+D9AFvB74DjeMEAucBXwF2b2KQAzuwB4EFgCjAXygaJT9uitOw74H+CfgRHA3wI/84cZIl/7T4FReL38v/Wf/xP/31y/p7w6ivcR6eP+Ps7138cNwCHn3MN4Pct/8ffbNhQwBXg3YvsngJsiHi8EPnTObfIf/wqY6Ne9yd9npM5tGikE/BdQAowH6oH/6GL7U9rFzMb7r30/UABMB7b423zPf7/TgXOAccBdbTt0zu0FmoDzkISgoB78RuIFS3PbExG9sHoz+5OIdX/pnPuDc67VOdfgnFvpnHvTf7wVL7Tm++teBzzvnHvN75V/E2jtpobP4w0nvODv6yVgA3BlxDr/5Zzb4Q85/BQvZPris/57arv9zn++CS8kJwHmnNvmnNvfw35ygbqIxz8BrjazTP/x5/znAHDO/dj/lnIS+DYwzf8W06ZDm0a+kHPukHPuZ865E865OrxAn09H3bXLzcDLzrknnHNN/r62+Mcc/hz4a+fcYX+/3wFu7LTfOv+9SgJQUA9+h4CRkQfjnHNl/hjuITr+DVRGbmhmF5nZ7/yv5rV449ptX73HRq7vnDvu768rJcD1kUEK/C+gMGKdAxH3TwDDo3+LAPzUOZcbcbvUr+u3eL3UB4APzOxhM8vuYT9H8IK97X29D2wDPumH9dX4QW1mSWb2XTPbaWZHgQp/s5ER++vQppHMLNPMfugPHx3FG+bJ7TSW3V27FAM7u9htAZAJbIxo61/7z0fKAmq6q03ii4J68FsNnASuiWLdzqdS/AnwHFDsnMsBHgLaZonsxwsLwAsdvOGPrlQCyzsF6TDn3HdPo6Y+c87d55ybBUzGGxK4o4d9b/XXidQ2/HEN8I4f3uD1rq8BLgdygLD/fORMmp7q/yre8MNFzrlsPhrmiWYmTiVwdhfPf4g3hDI5oq1znHPtH3xmNhZvGOXdLraXOKSgHuScczXA3cD/M7PrzGy4mYXMbDowrJfNs4DDzrkGM5uLF0xtngEW+we0UvEOWHb39/Q4Xo90od8LTTezBZEHznpQjTekclYU657CzOb43wxS8MbcG4AWf/EHXez3BU4dfngSb6z7L4gY9sBrn5N43yQy8YYY+iILL1RrzGwE8K0+bLsCuNzMPusf4Mw3s+nOuVbgP4F7zWwUeMcIzGxhxLYLgN+2HUiW+KegHgKcc/8C/A3wNeAgXkD9ELgT6OnHD7cD/2hmdXgHo34asc+3gb/EC679eEMGVd28fiVez/PreMFbider7fXvzzl3Am/s9g/+V/nSbla9wTrOoz7mB1U2XnAdAXbjheo9/jaPABf4+33Wf+4x4MrIWRL+mPZqoAx4KuI1H/P3uRd4B1jT2/vp5PtABl4veA3eEEVUnHN78Mb4vwocxjuQOM1ffCfwPrDGH1J5mY4HDm/G+3YkCcJ04QCRjszsO8BB59z3g66lv5nZFOBh51yfpjlKsBTUIiJxTkMfIiJxTkEtIhLnFNQiInGu2zOSnYmRI0e6cDgci12LiAxKGzdu/NA51/mHSUCMgjocDrNhw4ZY7FpEZFAys93dLdPQh4hInFNQi4jEOQW1iEici8kYtYgMTk1NTVRVVdHQ0ND7ytKl9PR0ioqKSElJiXobBbWIRK2qqoqsrCzC4TAWm8ttDmrOOQ4dOkRVVRUTJkyIejsNfYhI1BoaGsjPz1dInyYzIz8/v8/fSBTUItInCukzczrtFzdB3dDUwn++tovVO7u7SIiIyNAUN0GdFDL+8/VdPPhqV1cXEhGBpKQkpk+fzoUXXsgnP/lJampqelz/oYce4rHHHut1vzfddBNTp07l3nvv7adK+1fcHExMSQpx80Ul3PvyDnZVH+Osgr5eMk9EBruMjAy2bNkCQHl5OQ888ADf+MY3ul3/tttu63WfBw4cYNWqVeze3e0PA0/R3NxMcvLAxWfc9KgBbrqomJQk47HV0TeYiAxN8+bNY+/evQDs3LmTRYsWMWvWLC655BK2b98OwLe//W3uuce7oM+CBQu48847mTt3Lueeey6vv/46AB//+Mc5ePAg06dP5/XXX2fLli2UlpYydepUPv3pT3PkyJH27b/+9a8zf/58fvCDH7B+/XrKysqYNm0ac+fOpa6ujpaWFu644w7mzJnD1KlT+eEPf9gv7zVuetQAo7LSuWpKIc9srOJvF57H8LS4Kk9EItz932/zzr6j/brPC8Zm861PTu51vZaWFl555RX+7M/+DIBbb72Vhx56iIkTJ7J27Vpuv/12fvvb356yXXNzM+vWreOFF17g7rvv5uWXX+a5555j8eLF7T31qVOncv/99zN//nzuuusu7r77br7//e8DUFNTw6uvvkpjYyOTJk3iqaeeYs6cORw9epSMjAweeeQRcnJyWL9+PSdPnuTiiy/m4x//eJ+m4nUl7pLwlrIwz27Zx883VXHLvHDQ5YhIHKmvr2f69OlUVFQwa9YsrrjiCo4dO8aqVau4/vrr29c7ebLr6/Zee+21AMyaNYuKiopTltfW1lJTU8P8+d71jcvLyzvs94YbbgDg3XffpbCwkDlz5gCQnZ0NwIsvvsjWrVt55pln2vf33nvvDb6gnlGcy9SiHJatqmBJaYmmAonEqWh6vv2tbYy6traWxYsX88ADD7B06VJyc3Pbe8Q9SUtLA7yDks3NzX1+/WHDhgHeD1e6yibnHPfffz8LFy48ZdmZiKsxavDmGJbPC7Oz+jh/eF9T9UTkVDk5Odx3333cc889ZGRkMGHCBJ5++mnAC8s33njjtPebl5fXPn69fPny9t51pEmTJrFv3z7Wr18PQF1dHc3NzSxcuJAHH3yQpqYmAHbs2MHx48dPq5ZIcRfUAIunFZI/LJVHV1UEXYqIxKkZM2Ywbdo0nnzySVasWMEjjzzCtGnTmDx5Mr/85S9Pe7/Lli3jjjvuYOrUqWzZsoW77rrrlHVSU1N56qmn+PKXv8y0adO44ooraGho4Atf+AIXXHABM2fO5MILL+SLX/ziafXcO4vJVchnz57tzvTCAff85l0eWPk+r91xKcUjMvupMhE5E9u2beP8888PuoyE11U7mtlG59zsrtaPyx41wM2l4wmZsXyNpuqJyNAWt0FdmJPBwsmjeWp9JfWNLUGXIyISmLgNaoDyeWFq65t4dsveoEsREQlMXAf13AkjmDQmi2WrKojFWLqISCKI66A2M5aWhdl+oI51fzwcdDkiIoGI66AGuGb6OHIyUli2uiLoUkREAhH3QZ2RmsQNc4r5zdsfsK+mPuhyRCRAfT3N6WAR90ENsKS0hFbnWLFWU/VEhrK2n5C/9dZbjBgxggceeCBmr9XSEj+zzRIiqItHZHLZpNE8sa6Shqb4aTwRCU40pzldunQpt912G5dccgnnnnsuzz//PAAVFRVccsklzJw5k5kzZ7Jq1SoAVq5cyaWXXsrnPvc5pkyZAsCnPvUpZs2axeTJk3n44YfbX3/48OHceeedzJo1i8svv5x169axYMECzjrrLJ577rl+fa9xd1Km7iwtC/Pytg/4n637+cysoqDLEZFf/R0ceLN/9zlmCnziu72u1pfTnFZUVPDqq6+yc+dOLr30Ut5//31GjRrFSy+9RHp6Ou+99x433XQTbb+mXrduHW+99Vb7Ge9+/OMfM2LECOrr65kzZw6f+cxnyM/P5/jx4yxYsIDvfe97fPrTn+Yf/uEfeOmll3jnnXcoLy/n6quv7rdmiSqozeyvgS8ADngT+FPnXN8uo3uGLj4nn7MLhrFsdQXXzhyns+qJDEGnc5rTz372s4RCISZOnMhZZ53F9u3bmTBhAl/60pfYsmULSUlJ7Nixo339uXPndjgt6X333ccvfvELACorK3nvvffIz88nNTWVRYsWATBlyhTS0tJISUlhypQpXZ5C9Uz0GtRmNg7438AFzrl6M/spcCPwaL9W0nsdlJeFueuXb7O5soaZ4/MG8uVFpLMoer797XROc9q5U2dm3HvvvYwePZo33niD1tZW0tPT25e3ncoUvKGQl19+mdWrV5OZmcmCBQtoaPD6qCkpKe37DoVC7adQDYVC/XIipkjRjlEnAxlmlgxkAvv6tYooXTuziOFpySzTWfVEhrS+nOb06aefprW1lZ07d7Jr1y7OO+88amtrKSwsJBQKsXz58m4PHNbW1pKXl0dmZibbt29nzZo1A/L+Ous1qJ1ze4F7gD3AfqDWOfdi5/XM7FYz22BmG6qrq/u/UmB4WjLXzSrihTf3c7BuQEdeRCTORHua0/POO4/58+fziU98goceeoj09HRuv/12li1bRmlpKTt27OjQi460aNEimpubmTp1Kt/85jcpLS0dqLfXQa+nOTWzPOBnwA1ADfA08Ixz7vHutumP05x2Z1f1MT72b6/y15efy1cunxiT1xCRriXaaU6XLl3K4sWLue6664IupYNYnOb0cuCPzrlq51wT8HOg7IwrPU1nFQxn/rkFrFi7m8bm1qDKEBEZMNEE9R6g1MwyzRs5vwzYFtuyelZeVsLBupP8+u0DQZYhInHu0Ucfjbve9OmIZox6LfAMsAlval4IeLjHjWJswbmjKMnP1EFFkQDoTJZn5nTaL6pZH865bznnJjnnLnTOLXHOdX0t9gESChlLSkvYuPsIb+2tDbIUkSElPT2dQ4cOKaxPk3OOQ4cOdZgOGI2E+WViZ9fPLubfXtzBslUV/Ov104IuR2RIKCoqoqqqiljN7BoK0tPTKSrq26+rEzaoczJSuHbmOJ7eWMXfX3k+I4alBl2SyKCXkpLS4Vd7MjAS4qRM3bllXpjG5laeXL8n6FJERGImoYP6vDFZzDsrn8dX76a5RVP1RGRwSuigBigvC7OvtoGXtx0MuhQRkZhI+KC+/PxRjMvN0FQ9ERm0Ej6ok5NCfL60hNW7DvHugbqgyxER6XcJH9QAN84pJi05pAvgisigNCiCOm9YKldPG8svNu2l9kRT0OWIiPSrQRHU4B1UrG9q4emNlUGXIiLSrwZNUF84LofZJXksX7Ob1lb9vFVEBo9BE9Tg9ap3HzrByh2aqicig8egCupFF45hVFYaj67aHXQpIiL9ZlAFdUpSiJsvKuG1HdXsrD4WdDkiIv1iUAU1wE0XFZOSZCxfrV61iAwOgy6oR2Wlc9WUQp7ZWMWxk/17yXYRkSAMuqAG76DisZPN/HxTVdCliIicsUEZ1DPG5zGtKIdlqyp0JQoRSXiDMqjB61XvrD7O79//MOhSRETOyKAN6qumFpI/LFVn1RORhDdogzotOYmb5o7nle0HqTx8IuhyRERO26ANaoCbS8cTMmP5Gk3VE5HENaiDujAng0WTx/DU+krqG1uCLkdE5LQM6qAG76BibX0Tz27ZG3QpIiKnZdAH9ZxwHpPGZGmqnogkrEEf1GbG0rIw2w/Use6Ph4MuR0SkzwZ9UANcM30cORkpulSXiCSkIRHUGalJ3DinmN+8/QH7auqDLkdEpE+GRFADfL60hFbnWLFWU/VEJLEMmaAuHpHJZZNG88S6ShqaNFVPRBLHkAlqgKVlYQ4fb+R/tu4PuhQRkagNqaC++Jx8zhk1nGWrNVVPRBLHkApqM6N8Xglbq2rZXFkTdDkiIlEZUkENcO3MIrLSknVWPRFJGEMuqIelJXPd7CJeeHM/B+sagi5HRKRXUQW1meWa2TNmtt3MtpnZvFgXFktLSktoanE8sbYy6FJERHoVbY/6B8CvnXOTgGnAttiVFHtnFQxn/rkFrFi7m8bm1qDLERHpUa9BbWbZwJ8AjwA45xqdczUxrivmlpaFOVh3kl+/fSDoUkREehRNj/osoBr4LzPbbGY/MrNhnVcys1vNbIOZbaiuru73Qvvb/HMLCOdn6qCiiMS9aII6GZgJPOicmwEcB/6u80rOuYedc7Odc7MLCgr6ucz+FwoZS+aF2bj7CG/trQ26HBGRbkUT1FVAlXNurf/4GbzgTnjXzSoiIyVJvWoRiWu9BrVz7gBQaWbn+U9dBrwT06oGSE5GCtfOHMcv39jH4eONQZcjItKlaGd9fBlYYWZbgenAd2JW0QArLwvT2NzKk+v3BF2KiEiXogpq59wWf/x5qnPuU865I7EubKCcOzqLsrPzeXz1bppbNFVPROLPkPtlYlfKy8Lsq23g5W0fBF2KiMgpFNTAZZNGMS43g2WrdFEBEYk/CmogOSnE50tLWL3rEO8eqAu6HBGRDhTUvhvnFJOWHNIFcEUk7iiofXnDUrlm+lh+sWkvtSeagi5HRKSdgjpCeVmY+qYWnt6os+qJSPxQUEeYPDaHOeE8Hlu9m5ZWXapLROKDgrqTW+aF2XP4BK/uOBh0KSIigIL6FIsuHMPo7DQe1VQ9EYkTCupOUpJC3HxRCa/tqGZn9bGgyxERUVB35aa540lNCrF8tXrVIhI8BXUXCrLSuGpqIc9srOLYyeagyxGRIU5B3Y1b5pVw7GQzP99UFXQpIjLEKai7MWN8HtOKcli2qgLnNFVPRIKjoO5BeVmYndXH+f37HwZdiogMYQrqHlw1tZCRw1N1qS4RCZSCugdpyUncNHc8r2w/yJ5DJ4IuR0SGKAV1Lz530XhCZjy+VlP1RCQYCupeFOZksGjyGJ5aX0l9Y0vQ5YjIEKSgjkJ5WZja+iae3bI36FJEZAhSUEdhTjiP8wuzNVVPRAKhoI6CmbG0rITtB+pY+8fDQZcjIkOMgjpK10wfR25mCo/pUl0iMsAU1FFKT0nihtnF/ObtD9hXUx90OSIyhCio++DzpSU451ihqXoiMoAU1H1QPCKTy84fzRPrKmlo0lQ9ERkYCuo+WloW5vDxRp7fuj/oUkRkiFBQ91HZ2fmcM2q4puqJyIBRUPeRmVE+r4Q399ayubIm6HJEZAhQUJ+Ga2cWkZWWrLPqiciAUFCfhmFpyVw3u4gX3tzPwbqGoMsRkUFOQX2abpkXpqnF8ZO1e4IuRUQGOQX1aZowchgLzitgxdo9NDa3Bl2OiAxiCuozUF4WprruJL9++0DQpYjIIKagPgPzJxYQzs/UQUURiamog9rMksxss5k9H8uCEkkoZCyZF2bj7iO8tbc26HJEZJDqS4/6K8C2WBWSqK6fXURmahKPqlctIjESVVCbWRFwFfCj2JaTeLLTU7h25jiee2Mfh46dDLocERmEou1Rfx/4GtDt9AYzu9XMNpjZhurq6v6oLWGUzwvT2NzKUxsqgy5FRAahXoPazBYDB51zG3tazzn3sHNutnNudkFBQb8VmAgmjs6i7Ox8Hl+9m+YWTdUTkf4VTY/6YuBqM6sAngQ+ZmaPx7SqBFReFmZfbQMvb/sg6FJEZJDpNaidc3/vnCtyzoWBG4HfOuc+H/PKEszl549mXG6GDiqKSL/TPOp+khQylswrYc2uw2w/cDTockRkEOlTUDvnVjrnFseqmER3w+xi0pJDPLZal+oSkf6jHnU/yhuWyjXTx/KLTXupPdEUdDkiMkgoqPtZeVmY+qYWnt6oqXoi0j8U1P1s8tgc5oTzeGz1blpadakuETlzCuoYKC8Ls+fwCVa+ezDoUkRkEFBQx8DCyWMYk53OMh1UFJF+oKCOgZSkEDdfNJ7XdlSzs/pY0OWISIJTUMfIjXPHk5oUYrl61SJyhhTUMVKQlcZVUwt5ZmMVx042B12OiCQwBXUMlZeFOXaymZ9trAq6FBFJYArqGJpenMu04lyWra7AOU3VE5HTo6COsaVlJeyqPs7v3/8w6FJEJEEpqGPsyimFjByeqgvgishpU1DHWFpyEjfNHc8r2w+y59CJoMsRkQSkoB4AN19UQpIZy9dUBF2KiCQgBfUAGJOTzsILx/DU+krqG1uCLkdEEoyCeoAsLQtztKGZZ7fsDboUEUkwCuoBMrskj/MLs1m2SlP1RKRvFNQDxMxYWlbC9gN1rP3j4aDLEZEEoqAeQNdMH0duZoqm6olInyioB1B6ShI3zCnmxXc+YF9NfdDliEiCUFAPsCWlJTjnWLFWZ9UTkegoqAdYUV4ml58/mifWVdLQpKl6ItI7BXUAysvCHD7eyPNb9wddiogkAAV1AMrOzmfiqOGaqiciUVFQB8DMuKUszJt7a9m0pybockQkzimoA3LtjHFkpSfz2OqKoEsRkTinoA7IsLRkrp9VzAtv7udgXUPQ5YhIHFNQB2jJvBKaWhw/Wbsn6FJEJI4pqAM0YeQwFpxXwIq1e2hsbg26HBGJUwrqgJWXhamuO8mv3tJUPRHpmoI6YPMnFjBh5DAeW61fKopI1xTUAQuFjCWlJWzcfYS39tYGXY6IxCEFdRy4bnYRmalJPKqz6olIFxTUcSA7PYVrZ47juTf2cejYyaDLEZE4o6COE+XzwjQ2t/Lk+sqgSxGRONNrUJtZsZn9zsy2mdnbZvaVgShsqJk4OouLz8lnxZrdNLdoqp6IfCSaHnUz8FXn3PlAKfCXZnZBbMsamsrnhdlX28DL2z4IuhQRiSO9BrVzbr9zbpN/vw7YBoyLdWFD0WXnj2ZcboYOKopIB30aozazMDADWNvFslvNbIOZbaiuru6n8oaWpJCxZF4Ja3YdZvuBo0GXIyJxIuqgNrPhwM+Av3LOnZIizrmHnXOznXOzCwoK+rPGIeWG2cWkJYdYtko/gBERT1RBbWYpeCG9wjn389iWNLTlDUvlU9PH8ezmvdSeaAq6HBGJA9HM+jDgEWCbc+7fY1+SlJeFqW9q4emNmqonItH1qC8GlgAfM7Mt/u3KGNc1pF0wNpu54RE8tno3La26VJfIUBfNrI/fO+fMOTfVOTfdv70wEMUNZbeUlbDn8AlWvnsw6FJEJGD6ZWKcWjh5DGOy0zVVT0QU1PEqJSnEzReN5/X3PmRn9bGgyxGRACmo49hNF40nNSnEcp2rWmRIU1DHsZHD01g8tZBnNlZx7GRz0OWISEAU1HGuvCzMsZPN/GxjVdCliEhAFNRxblpxLtOKc1m2uoJWTdUTGZIU1AlgaVkJu6qP84edHwZdiogEQEGdAK6cUsjI4aks01Q9kSFJQZ0A0pKT+Nzc8byy/SB7Dp0IuhwRGWAK6gRxc2kJSWYsX1MRdCkiMsAU1AlidHY6Cy8cw1PrKznRqKl6IkOJgjqBLC0Lc7ShmWc37wu6FBEZQArqBDK7JI8LCrN5bHUFzmmqnshQoaBOIGbG0rIw2w/UsfaPh4MuR0QGiII6wVw9fSy5mSmaqicyhCioE0x6ShI3zCnmxXc+YF9NfdDliMgAUFAnoCWlJTjneHyNzqonMhQoqBNQUV4ml58/mifXV9LQ1BJ0OSISYwrqBLW0LMzh44384JX32FV9TLNARAax5KALkNMz7+x85k4YwYMrd/Lgyp3kZqYwoziXGePzmDHeO+NednpK0GWKSD9QUCcoM+OJPy9lZ/UxNu0+wuY9NWyuPMLKHdU4B2YwcdRwZhR7wT2zJI9zCoYTClnQpYtIH1ksvjLPnj3bbdiwod/3K7072tDEG5U1bN5Tw6Y9XoDX1jcBkJWWzLTiXGaO93re04tzyRuWGnDFIgJgZhudc7O7WqYe9SCTnZ7CJRMLuGRiAQDOOf744fEOwf0fv3uftmsQTBg5jBl+cM8ozmXSmCySk3ToQiSeqEc9BB0/2czWqlo2V/pDJnuO8OGxRgAyUpKYWpTTPtY9Y3wuo7LSA65YZPBTj1o6GJaWzLyz85l3dj7g9bqrjtS397g37znCj17fRbPf7S7Ky2DG+Lz2IZMLCrNJTVavW2SgKKgFM6N4RCbFIzK5Zvo4ABqaWnh7X237kMmGisP89xveWftSk0NMGZfTYZbJ2NyMIN+CyKCmoQ+J2v7a+vYe9+Y9NWzdW0tjcysAY7LT24dKZo7P48JxOaSnJAVcsUji0NCH9IvCnAwKp2Rw5ZRCABqbW9m2/6gX3JVez/tXbx0AIDlkXDA2mxnF3tTAGcV5FI/IwEzTA0X6Kr561A9eDMnpkF0I2eMgy/83uxCyx0LWWEjRga14Vl13ki2VXq97054jbK2q5USj9zP3kcNTmV780UHKaUW5DEtTX0EEEqVH3doKhdPg6F6o3gE7V0Jj3anrZeRFhPjYj25ZbfcLIT3X+8WHDLiCrDSuuGA0V1wwGoDmllZ2fHDsowOVlUd4edsHAIQMzhuT7QW33/OekD9MP8oR6SS+etSdNRyFuv1wdJ93q9sHR/dH3N8Hx6tP3S45o1OId9EzHz4KQhpDDcKR441sqappH+/eUllDXYN3HcicjBSmF3vj3G0/hc/J0E/hZfDrqUcd30EdjeZGL8zr9nu98Q5B3nZ/P7Q2ddzOkiBrTM89cw21DIjWVsfO6mPtPe5Nu2vYcbCOtj/Nc0YNb58aOGN8LhNHZZGkXrcMMoM7qKPR2gonPuy5Z350HzQeO3XbjBE998yzx0J6joZa+lldQxNbq2q985j4Y95HTngftsPTkplWnNN+HpMZ4/MYoZ/CS4JTUEerfailq575Xm9ZV0MtKZk998yzx8GwAg21nAHnHBWHTrRPDdy05wjbD9TR4v8oJ5yf2eFHOeeNySJFP4WXBKKg7k/NJ6HuQKfe+P5T77c2d9zOkvwwL+zUM484MJpVqKGWPjjR2MybVbXe1MDdR9i0p4YPj50EID0lxNRxucwoyWVGsRfgo7LVthK/zjiozWwR8AMgCfiRc+67Pa0/qIM6Gu1DLXs79sY73O9mqCUz3++NF3bdM88q1FBLN5xz7K2p73ACqrf31dLU4v2Nj8vNYPyITFKSQ6QmGcmhECnJIVKSjNSkEClJIZIj7qckhUhJ9h4nh8xfN9TNuv7ykLdN5HopSUZy+2MjKWSaTy6nOKOgNrMkYAdwBVAFrAducs690902Qz6oo9Vw9NSeeWSQH93nBX5nKcM+6pmnZXuhHUoCC3k998j77cv859qXhTqtd5rLIp9vX9bFNj0tO6PX6mrZRyHo/RT+ox/lVB89SWNLK82trTQ1O5paWmlsaaWppZWmFuf/691vG1bpb2Z4AX5K+Jv/AeB9kHz0YeGvmxTdB0tKKOKDw993ctKpHx7dfbBErhsKgWHtTWrt78HaH7cv04fPGTnTedRzgfedc7v8nT0JXAN0G9QSpfRs7zZqUvfrNJ/sujfeNpulZg+4FmhtAdfq3XetXq++/X6UywaNjz6c0i3ErFASsyIDvsOq5qVNsn/jo7BxfvC09WXaYtv5/+nwGIt43uH4aNuutnOd9utagGZobVvXX9YauU7EtriO60buz1vcMTQ7f+RELm+73+jfOuu8r9NiPT6Mched3lMPO7Eu7vW8796eiG7x8eQcJn9jVVSv2RfRBPU4oDLicRVwUeeVzOxW4FaA8ePH90txAiSnQV7Yu8Vaa2unQG/pFO5RLOuPD4wzeq2ulrmIfURGVqf46rSsvfd4yrJO257uslMe9lRP7/ttdc57q67VawrnaHEO57xlra3QisO1+o9d5H387dvWdX74u1Ne3nV63S7L6f5ddfiQ++ipLnbQ5et1v7Cr9Vw3O+lynz28nou818O+WlKyutrzGYsmqLv68Di17Z17GHgYvKGPM6xLghAK4V3vOH5+sCrRa/uuoLlFg08085eqgOKIx0XAvtiUIyIinUUT1OuBiWY2wcxSgRuB52JbloiItOn1O65zrtnMvgT8Bu9b1Y+dc2/HvDIREQGiHIx0zr0AvBDjWkREpAv6ja2ISJxTUIuIxDkFtYhInFNQi4jEuZicPc/MqoHdp7n5SKCLE1wETnX1jerqG9XVN4OxrhLnXEFXC2IS1GfCzDZ0d2KSIKmuvlFdfaO6+mao1aWhDxGROKegFhGJc/EY1A8HXUA3VFffqK6+UV19M6TqirsxahER6Sgee9QiIhJBQS0iEucCCWozW2Rm75rZ+2b2d10sNzO7z1++1cxmxkldC8ys1sy2+Le7BqiuH5vZQTN7q5vlQbVXb3UF1V7FZvY7M9tmZm+b2Ve6WGfA2yzKuga8zcws3czWmdkbfl13d7FOEO0VTV2B/I35r51kZpvN7PkulvVvezn/8jsDdcM7VepO4CwgFXgDuKDTOlcCv8K7ukwpsDZO6loAPB9Am/0JMBN4q5vlA95eUdYVVHsVAjP9+1l4F2eOh7+xaOoa8Dbz22C4fz8FWAuUxkF7RVNXIH9j/mv/DfCTrl6/v9sriB51+8VynXONQNvFciNdAzzmPGuAXDMrjIO6AuGcew043MMqQbRXNHUFwjm33zm3yb9fB2zDu/ZnpAFvsyjrGnB+GxzzH6b4t86zDIJor2jqCoSZFQFXAT/qZpV+ba8ggrqri+V2/mONZp0g6gKY538V+5WZTY5xTdEKor2iFWh7mVkYmIHXG4sUaJv1UBcE0Gb+1/gtwEHgJedcXLRXFHVBMH9j3we+xkcXg++sX9sriKCO5mK5UV1Qt59F85qb8H6PPw24H3g2xjVFK4j2ikag7WVmw4GfAX/lnDvaeXEXmwxIm/VSVyBt5pxrcc5Nx7sm6lwzu7DTKoG0VxR1DXh7mdli4KBzbmNPq3Xx3Gm3VxBBHc3FcoO4oG6vr+mcO9r2Vcx5V71JMbORMa4rGnF5AeIg28vMUvDCcIVz7uddrBJIm/VWV9B/Y865GmAlsKjTokD/xrqrK6D2uhi42swq8IZIP2Zmj3dap1/bK4igjuZiuc8Bt/hHTkuBWufc/qDrMrMxZmb+/bl47XcoxnVFI4j26lVQ7eW/5iPANufcv3ez2oC3WTR1BdFmZlZgZrn+/QzgcmB7p9WCaK9e6wqivZxzf++cK3LOhfFy4rfOuc93Wq1f2yuqayb2J9fNxXLN7DZ/+UN412e8EngfOAH8aZzUdR3wF2bWDNQDNzr/EG8smdkTeEe3R5pZFfAtvAMrgbVXlHUF0l54PZ4lwJv++CbA14HxEbUF0WbR1BVEmxUCy8wsCS/ofuqcez7o/yejrCuov7FTxLK99BNyEZE4p18miojEOQW1iEicU1CLiMQ5BbWISJxTUIuIxDkFtYhInFNQi4jEuf8PE7CDoiCSls0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "fig = plt.figure()\n",
    "ax = plt.subplot(111)\n",
    "\n",
    "ax.plot(var_reinforce, label='Reinforce')\n",
    "ax.plot(var_reparam, label='Reparam')\n",
    "\n",
    "plt.title('Gradient Est(variance)')\n",
    "ax.legend()\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
